BayesNAS: A Bayesian Approach for Neural Architecture Search

One-Shot Neural Architecture Search (NAS) is a promising method to significantly reduce search time without any separate training. It can be treated as a Network Compression problem on the architecture parameters from an over-parameterized network. However, there are two issues associated with most one-shot NAS methods. First, dependencies between a node and its predecessors and successors are often disregarded which result in improper treatment over zero operations. Second, architecture parameters pruning based on their magnitude is questionable. In this paper, we employ the classic Bayesian learning approach to alleviate these two issues by modeling architecture parameters using hierarchical automatic relevance determination (HARD) priors. Unlike other NAS methods, we train the over-parameterized network for only one epoch then update the architecture. Impressively, this enabled us to find the architecture on CIFAR-10 within only 0.2 GPU days using a single GPU. Competitive performance can be also achieved by transferring to ImageNet. As a byproduct, our approach can be applied directly to compress convolutional neural networks by enforcing structural sparsity which achieves extremely sparse networks without accuracy deterioration.Comment: International Conference on Machine Learning 201

Do humans and machines have the same eyes? Human-machine perceptual differences on image classification

Trained computer vision models are assumed to solve vision tasks by imitating human behavior learned from training labels. Most efforts in recent vision research focus on measuring the model task performance using standardized benchmarks. Limited work has been done to understand the perceptual difference between humans and machines. To fill this gap, our study first quantifies and analyzes the statistical distributions of mistakes from the two sources. We then explore human vs. machine expertise after ranking tasks by difficulty levels. Even when humans and machines have similar overall accuracies, the distribution of answers may vary. Leveraging the perceptual difference between humans and machines, we empirically demonstrate a post-hoc human-machine collaboration that outperforms humans or machines alone.Comment: Paper under revie

A Survey for High-redshift Gravitationally Lensed Quasars and Close Quasars Pairs. I. the Discoveries of an Intermediately-lensed Quasar and a Kpc-scale Quasar Pair at z∼5z\sim5

We present the first results from a new survey for high-redshift $(z\gtrsim5)$ gravitationally lensed quasars and close quasar pairs. We carry out candidate selection based on the colors and shapes of objects in public imaging surveys, then conduct follow-up observations to confirm the nature of high-priority candidates. In this paper, we report the discoveries of J0025--0145 ($z=5.07$) which we identify as an {intermediately-lensed quasar, and J2329--0522 ($z=4.85$) which is a kpc-scale close quasar pair. The {\em Hubble Space Telescope (HST)} image of J0025--0145 shows a foreground lensing galaxy located 0\farcs6 away from the quasar. However, J0025--0145 does not exhibit multiple lensed images of the quasar, and we identify J0025--0145 as an intermediate lensing system (a lensing system that is not multiply imaged but has a significant magnification). The spectrum of J0025--0145 implies an extreme Eddington ratio if the quasar luminosity is intrinsic, which could be explained by a large lensing magnification. The {\em HST} image of J0025--0145 also indicates a tentative detection of the quasar host galaxy in rest-frame UV, illustrating the power of lensing magnification and distortion in studies of high-redshift quasar host galaxies. J2329--0522 consists of two resolved components with significantly different spectral properties, and a lack of lensing galaxy detection under sub-arcsecond seeing. We identify it as a close quasar pair, which is the highest confirmed kpc-scale quasar pair to date. We also report four lensed quasars and quasar pairs at $2<z<4$, and discuss possible improvements to our survey strategy.Comment: 23 pages, 10 figures, 6 tables. Accepted by the Astronomical Journa

Effects of Electrode Off Centre on Trapped Thickness-Shear Modes in Contoured AT-Cut Quartz Resonators

We investigated thickness-shear vibrations of a contoured, AT-cut quartz resonator with a pair of electrodes displaced from the resonator centre. The scalar differential equations by Stevens and Tiersten for thickness-shear vibrations of electroded and unelectroded quartz plates were employed. Based on the variational formulation of the scalar differential equations established in a previous paper and the variation-based Ritz method with trigonometric functions as basis functions, free vibration resonance frequencies and trapped thickness-shear modes were obtained. The effects of the electrode off centre on resonance frequencies and mode shapes were examined. When the electrode off centre is about one hundredth of the electrode length, the relative frequency shift is of the order of one part per million, significant in certain resonator design and applications. The electrode off centre also causes the loss of symmetry of modes, which has an adverse effect on resonator frequency stability under a normal acceleration

Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers

We propose to address quadrupedal locomotion tasks using Reinforcement Learning (RL) with a Transformer-based model that learns to combine proprioceptive information and high-dimensional depth sensor inputs. While learning-based locomotion has made great advances using RL, most methods still rely on domain randomization for training blind agents that generalize to challenging terrains. Our key insight is that proprioceptive states only offer contact measurements for immediate reaction, whereas an agent equipped with visual sensory observations can learn to proactively maneuver environments with obstacles and uneven terrain by anticipating changes in the environment many steps ahead. In this paper, we introduce LocoTransformer, an end-to-end RL method for quadrupedal locomotion that leverages a Transformer-based model for fusing proprioceptive states and visual observations. We evaluate our method in challenging simulated environments with different obstacles and uneven terrain. We show that our method obtains significant improvements over policies with only proprioceptive state inputs, and that Transformer-based models further improve generalization across environments. Our project page with videos is at https://RchalYang.github.io/LocoTransformer .Comment: Our project page with videos is at https://RchalYang.github.io/LocoTransforme

When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks

Recent advances in adversarial attacks uncover the intrinsic vulnerability of modern deep neural networks. Since then, extensive efforts have been devoted to enhancing the robustness of deep networks via specialized learning algorithms and loss functions. In this work, we take an architectural perspective and investigate the patterns of network architectures that are resilient to adversarial attacks. To obtain the large number of networks needed for this study, we adopt one-shot neural architecture search, training a large network for once and then finetuning the sub-networks sampled therefrom. The sampled architectures together with the accuracies they achieve provide a rich basis for our study. Our "robust architecture Odyssey" reveals several valuable observations: 1) densely connected patterns result in improved robustness; 2) under computational budget, adding convolution operations to direct connection edge is effective; 3) flow of solution procedure (FSP) matrix is a good indicator of network robustness. Based on these observations, we discover a family of robust architectures (RobNets). On various datasets, including CIFAR, SVHN, Tiny-ImageNet, and ImageNet, RobNets exhibit superior robustness performance to other widely used architectures. Notably, RobNets substantially improve the robust accuracy (~5% absolute gains) under both white-box and black-box attacks, even with fewer parameter numbers. Code is available at https://github.com/gmh14/RobNets.Comment: CVPR 2020. First two authors contributed equall

Exploring Reionization-Era Quasars IV: Discovery of Six New z≳6.5z \gtrsim 6.5 Quasars with DES, VHS and unWISE Photometry

This is the fourth paper in a series of publications aiming at discovering quasars at the epoch of reionization. In this paper, we expand our search for $z\sim 7$ quasars to the footprint of the Dark Energy Survey (DES) Data Release One (DR1), covering $\sim 5000$ deg$^2$ of new area. We select $z\sim 7$ quasar candidates using deep optical, near-infrared (near-IR) and mid-IR photometric data from the DES DR1, the VISTA Hemisphere Survey (VHS), the VISTA Kilo-degree Infrared Galaxy (VIKING) survey, the UKIRT InfraRed Deep Sky Surveys -- Large Area Survey (ULAS) and the unblurred coadds from the Wide-field Infrared Survey Explore ($WISE$) images (unWISE). The inclusion of DES and unWISE photometry allows the search to reach $\sim$ 1 magnitude fainter, comparing to our $z \gtrsim 6.5$ quasar survey in the northern sky (Wang et al. 2018). We report the initial discovery and spectroscopic confirmation of six new luminous quasars at $z>6.4$, including an object at $z=7.02$, the fourth quasar yet known at $z>7$, from a small fraction of candidates observed thus far. Based on the recent measurement of $z \sim 6.7$ quasar luminosity function using the quasar sample from our survey in the northern sky, we estimate that there will be $\gtrsim$ 55 quasars at $z > 6.5$ at $M_{1450} < -24.5$ in the full DES footprint.Comment: 8 pages, 3 figures, submitted to A